

Application No. 


Applicant(s) 


Notice of Allowability 


10/815,354 


CHIU ET AL 


Examiner 


Art Unit 






EDWARD PARK 


2624 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address- 

All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not included 
herewith {or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication will be mailed in due course. THIS 
NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiative 
of the Office or upon petition by the applicant. See 37 CFR 1.313 and MPEP 1308. 

1 . ^ This communication is responsive to IDS received on 1 1/9/09 . 

2. ^ The allowed claim(s) is/are 1, 3-6, 10-15, 19-20, 24-27 (to be re-numbered as 1-17) . 

3. □ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a) □ All b) □ Some* c) □ None of the: 

1. □ Certified copies of the priority documents have been received. 

2. □ Certified copies of the priority documents have been received in Application No. . 

3. □ Copies of the certified copies of the priority documents have been received in this national stage application from the 

International Bureau (PCT Rule 17.2(a)). 
* Certified copies not received: . 

Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirements 
noted below. Failure to timely comply will result in ABANDONMENT of this application. 
THIS THREE-MONTH PERIOD IS NOT EXTENDABLE 

4. □ A SUBSTITUTE OATH OR DECLARATION must be submitted. Note the attached EXAMINER'S AMENDMENT or NOTICE OF 

INFORMAL PATENT APPLICATION (PTO-152) which gives reason(s) why the oath or declaration is deficient. 

5. □ CORRECTED DRAWINGS ( as "replacement sheets") must be submitted. 

(a) □ including changes required by the Notice of Draftsperson's Patent Drawing Review ( PTO-948) attached 

1) □ hereto or 2) □ to Paper No./Mail Date . 

(b) □ including changes required by the attached Examiner's Amendment / Comment or in the Office action of 

Paper No./Mail Date . 

Identifying indicia such as the application number (see 37 CFR 1.84(c)) should be written on the drawings in the front (not the back) of 
each sheet. Replacement sheet(s) should be labeled as such in the header according to 37 CFR 1.121(d). 

6. □ DEPOSIT OF and/or INFORMATION about the deposit of BIOLOGICAL MATERIAL must be submitted. Note the 

attached Examiner's comment regarding REQUIREMENT FOR THE DEPOSIT OF BIOLOGICAL MATERIAL. 



Attachment(s) 

1 . □ Notice of References Cited (PTO-892) 

2. □ Notice of Draftperson's Patent Drawing Review (PTO-948) 

3. M Information Disclosure Statements (PTO/SB/08), 

Paper No./Mail Date 3/24/05. 11/9/09 

4. □ Examiner's Comment Regarding Requirement for Deposit 

of Biological Material 



/Edward Park/ 
Examiner, Art Unit 2624 



5. □ Notice of Informal Patent Application 

6. M Interview Summary (PTO-413), 

Paper No./Mail Date 10/29/09 . 

7. M Examiner's Amendment/Comment 

8. ^ Examiner's Statement of Reasons for Allowance 

9. □ Other . 



U.S. Patent and Trademark Office 
PTOL-37 (Rev. 08-06) 



Notice of Allowability 



Part of Paper No./Mail Date 20091 110 



Application/Control Number: 10/815,354 Page 2 

Art Unit: 2624 

EXAMINER'S STATEMENT OF REASONS FOR ALLOWANCE 



Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.1 14. Applicant's submission filed on 1 1/9/09 has been entered. 



EXAMINER'S AMENDMENT 

2. An examiner's amendment to the record appears below. Should the changes and/or 
additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 
1.312. To ensure consideration of such an amendment, it MUST be submitted no later than the 
payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview with 
Anthony G. Craig (Reg #: 50, 342) on 10/29/09. 

The application has been amended as follows: 
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1 . (Currently Amended): A computer implemented method implemented within a 
computer system including memory and CPU for generating a highly condensed visual summary 
of video regions, comprising: 

utilizing the memory and CPU for determining a dominant group in each of a plurality of 
video segments; 

utilizing the memory and CPU for determining a key frame in each of the video 

utilizing the memory and CPU for defining a germ in each of a plurality of images, the 
germ containing a region of interest associated with each dominant group in each of the video 
segments, wherein the video segment less the germ defines a support in each of the video 

utilizing the memory and CPU for defining a support in each of the video segments, wherein the 
support is the video segment less the germ; 

utilizing the memory and CPU for separating the germ from the video segments; 
utilizing the memory and CPU for laying out the germs on a canvas , wherein there is no 
more than one germ for every video segment , wherein the canvas is partitioned into disjoint areas 
corresponding to the shape of the germs, wherein the shape of the germs disjoint areas is defined 
by a Voronoi algorithm; and 

utilizing the memory and CPU for filling in the space of the canvas inside each disjoint 
area not occupied by its corresponding germ between the disjoint areas corresponding to the 
shape of the germs , wherein filling in the space of the canvas between the germs includes laying 
out one or more portions of the supports by assigning a pixel value of a point in the space from 
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the same value as the corresponding pixel of the germ's support when this point is nearest the 
germ, and only when the germ's support does not encompass the point assigning pixel values of 
a support of a neighboring germ based on a distance from the point to the neighboring germ , 
wherein a point in the space is only assigned a background value if no support includes the 
germ's support docs not include the point, wherein the canvas generated is to generate a highly 
condensed visual summary of the plurality of video segments. 

2. (Cancelled) 

3. (Currently Amended): The method of claim 1 wherein defining a germ includes: 
defining a two dimensional shape that encompasses the projection of a the dominant 
group onto the key frame. 

4. (Original): The method of claim 3 wherein the two dimensional shape is a rectangle. 

5. (Original): The method of claim 3 wherein laying out the germs includes: 
determining a scale factor to be applied to every germ such that the germs are scaled to 
the maximum size that fits into the canvas. 

6. (Original): The method of claim 3 wherein laying out the germs includes: 
placing the germs in rows, wherein each row has a height according to the longest germ 
in the particular row. 

7-9. (Cancelled) 

10. (Currently Amended): A computer implemented method implemented within a 
computer system including memory and CPU for generating a highly condensed visual summary 
of video regions, comprising: 

utilizing the memory and CPU for determining a germ in each of a plurality of images, 
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the germ containing a region of interest; 

utilizing the memory and CPU for defining , wherein the video region less the germ 
defines a support in each of the video segments regions wherein the support is the video segment 
less the germ ; 

utilizing the memory and CPU for separating the germ from the video segments; 

utilizing the memory and CPU for laying out the germs on a canvas, wherein the germs 

are laid out in irregular two dimensional shapes on the canvas; 

utilizing the memory and CPU for defining a space between the germs; and 

utilizing the memory and CPU for filling in the space of the canvas between the irregular 

two dimensional shape germs by laying out one or more parts of the support , wherein filling in 

the space of the canvas between the irregular two dimensional shape germs includes laying out 

one or more portions of the supports by assigning the same value as the corresponding pixel of 

the germ's support when this point is nearest the germ, and only when the germ's support does 

not encompass the point assigning a pixel value of a point in the space from pixel values of a 

support of a neighboring germ based on a distance from the point to the neighboring germ, 

wherein a point between the irregular two dimensional shape germs is assigned an average value 

of nearby pixels point values only if no support includes the point , to generate wherein the 

canvas generated is a highly condensed visual summary of the plurality of video segments 

11. (Previously Presented): The method of claim 10 wherein determining a germ 
includes: detecting a face in each of the plurality of images. 
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12. (Previously Presented): The method of claim 10 wherein determining a germ 
includes: receiving user input, the user input associated with a part of an image. 

13. (Previously Presented): The method of claim 10 wherein determining a germ 
includes: using an algorithm to determine the regions of interest of an image based on one or 
more methods selected from the group consisting of a face-detection algorithm, an object 
detection algorithms and user input. 

14. (Previously Presented): The method of claim 10 wherein laying out the germs 
includes: determining a scale factor to be applied to every germ such that the germs are scaled to 
the maximum size that fits into the canvas. 

15. (Previously Presented): The method of claim 10 wherein laying out the germs 
includes: placing the germs in rows, wherein each row has a height according to the longest germ 
in the particular row. 

16-18. (Cancelled) 

19. (Previously Presented): The method of claim 1 wherein defining a germ includes: 
detecting a face in each of the plurality of images. 

20. (Previously Presented): The method of claim 1 wherein defining a germ includes: 
using an algorithm to determine a region of interest of an image. 

21 -23. (Cancelled) 

24. (Previously Presented): A computer implemented method implemented within a 
computer system including memory and CPU for generating a highly condensed visual summary 
of video regions, comprising: 

utilizing the memory and CPU for determining a dominant group in each of a plurality of 
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video segments, wherein the dominant group includes a face; 

utilizing the memory and CPU for determining a key frame in each of the video 

segments; 

utilizing the memory and CPU for defining a germ associated with each dominant group 
in each of the video segments, wherein the germ is the x-y projection of the dominant group 
including the face onto the keyframe; 

utilizing the memory and CPU for separating the germ from the video segments; 

utilizing the memory and CPU for laying out the germs on a canvas, wherein the canvas 

is partitioned into disjoint areas corresponding to the germs, wherein the shape of the disjoint 

areas is defined using one or more algorithm selected from the group consisting of the distances 

between the germs, the distance between the face and the germ and the distance between two or 

more faces and the germ; and 

utilizing the memory and CPU for filling in the space of the canvas between the disjoint 
areas corresponding to the germs, wherein filling in the space of the canvas between the germs 
includes laying out one or more portions of the supports by assigning a pixel value of a point in 
the space from the same value as the corresponding pixel of the germ's support when this point 
is nearest the germ, and only when the germ's support does not encompass the point assigning 
pixel values of a support of a neighboring germ based on a distance from the point to the 
neighboring germ, wherein a point between the irregular two dimensional shape germs is 
assigned an average value of nearby point values only if no support includes the point, wherein 
the canvas generated is a highly condensed visual summary of the plurality of video segments. 
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25. (Previously Presented): A computer implemented method implemented within a 
computer system including memory and CPU for generating a highly condensed visual summary 
of video regions, comprising: 

utilizing the memory and CPU for determining a germ in each of a plurality of images, 
the germ containing a region of interest; 

utilizing the memory and CPU for defining a support in each of the video segments, 
wherein the support is the video segment less the germ; 

utilizing the memory and CPU for separating the germ from the video segments; 
utilizing the memory and CPU for laying out the germs on a canvas, wherein the canvas 
is partitioned into disjoint areas corresponding to the shape of the germs; 
utilizing the memory and CPU for computing boundary curves between the germs, 
wherein the boundary curves between the germs are defined using one or more algorithm 
selected from the group consisting of the distance between a point and the closest border of the 
germ, the distance between a point and the center of a germ and the distance between a point and 
the size of the germ; 

utilizing the memory and CPU for defining a space between the boundary curves; and 
utilizing the memory and CPU for filling in the space of the canvas, wherein filling in the 
space of the canvas includes laying out one or more portions of the supports by assigning a pixel 
value of a point in the space from the same value as the corresponding pixel of the germ's 
support when this point is nearest the germ, and only when the germ's support does not 
encompass the point assigning pixel values of a support of a neighboring germ based on a 
distance from the point to the neighboring germ, wherein a point between the boundary curves is 
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assigned an average value of nearby point values only if no support includes the point, to 
generate a highly condensed visual summary of the plurality of video segments. 

26. (Previously Presented): The method of claim 3 wherein the two dimensional shape is 
irregular. 

27. (Previously Presented): The computer implemented method of claim 25 wherein the 
germs are laid out in irregular two dimensional shapes on the canvas. 



Allowable Subject Matter 

3. Claims 1, 3-6, 10-15, 19-20, 24-27 (to be re-numbered as 1-17) are allowed. 

4. The following is an examiner's statement of reasons for allowance: 

Examiner notes that the IDS filed on 1 1/9/09 has been considered and does not affect 
Examiner's stance on the patentability of the application. 

Regarding claim 1 , the most relevant prior art of record, Bae reference, teaches utilizing the 

memory and CPU for defining a germ in each of a plurality of images, the 

germ containing a region of interest; utilizing the memory and CPU for defining a support in 

each of the video segments, wherein the support is the video segment less the germ; utilizing the 

memory and CPU for separating the germ from the video segments (see Final Rejection on 

8/24/09). 
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Applicant's claimed invention distinguishes over the Bae reference by utilizing the 
memory and CPU for laying out the germs on a canvas, wherein there is no 
more than one germ for every video segment, wherein the canvas is partitioned into disjoint areas 
corresponding to the shape of the germs, wherein the shape of the disjoint areas is defined by a 
Voronoi algorithm; and utilizing the memory and CPU for filling in the space of the canvas 
between the disjoint areas corresponding to the shape of the germs, wherein filling in the space 
of the canvas between the germs includes laying out one or more portions of the supports by 
assigning a pixel value of a point in the space from the same value as the corresponding pixel of 
the germ's support when this point is nearest the germ, and only when the germ's support does 
not encompass the point assigning pixel values of a support of a neighboring germ based on a 
distance from the point to the neighboring germ, wherein a point in the space is only assigned a 
background value if no support includes the point, to generate a highly condensed visual 
summary of the plurality of video segments. 

Regarding claim 10, the most relevant prior art of record, Yu reference, teaches utilizing the 
memory and CPU for determining a germ in each of a plurality of images, the germ containing a 
region of interest; utilizing the memory and CPU for defining^ a support in each of the video 
segments wherein the support is the video segment less the germ; 

utilizing the memory and CPU for separating the germ from the video segments; utilizing the 
memory and CPU for laying out the germs on a canvas, wherein the germs are laid out in 
irregular two dimensional shapes on the canvas; utilizing the memory and CPU for defining a 
space between the germs (see Final Rejection on 8/24/09). 
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Applicant's claimed invention distinguishes over the Bae reference by utilizing the 
memory and CPU for filling in the space of the canvas between the irregular 
two dimensional shape germs, wherein filling in the space of the canvas between the irregular 
two dimensional shape germs includes laying out one or more portions of the supports by 
assigning the same value as the corresponding pixel of the germ's support when this point is 
nearest the germ, and only when the germ's support does not encompass the point assigning a 
pixel value of a point in the space from pixel values of a support of a neighboring germ based on 
a distance from the point to the neighboring germ, wherein a point between the irregular two 
dimensional shape germs is assigned an average value of nearby point values only if no support 
includes the point, to generate a highly condensed visual summary of the plurality of video 
segments. 

Regarding claim 24, the most relevant prior art of record, Yu with Li combination, teaches 
utilizing the memory and CPU for determining a dominant group in each of a plurality of video 
segments, wherein the dominant group includes a face; utilizing the memory and CPU for 
determining a key frame in each of the video segments; utilizing the memory and CPU for 
defining a germ associated with each dominant group in each of the video segments, wherein the 
germ is the x-y projection of the dominant group including the face onto the keyframe; utilizing 
the memory and CPU for separating the germ from the video segments (see Final Rejection on 
8/24/09). 

Applicant's claimed invention distinguishes over the Yu with Li combination by utilizing 
the memory and CPU for laying out the germs on a canvas, wherein the canvas 
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is partitioned into disjoint areas corresponding to the germs, wherein the shape of the disjoint 
areas is defined using one or more algorithm selected from the group consisting of the distances 
between the germs, the distance between the face and the germ and the distance between two or 
more faces and the germ; and utilizing the memory and CPU for filling in the space of the canvas 
between the disjoint areas corresponding to the germs, wherein filling in the space of the canvas 
between the germs includes laying out one or more portions of the supports by assigning a pixel 
value of a point in the space from the same value as the corresponding pixel of the germ's 
support when this point is nearest the germ, and only when the germ's support does not 
encompass the point assigning pixel values of a support of a neighboring germ based on a 
distance from the point to the neighboring germ, wherein a point between the irregular two 
dimensional shape germs is assigned an average value of nearby point values only if no support 
includes the point, wherein the canvas generated is a highly condensed visual summary of the 
plurality of video segments. 

Regarding claim 25, the most relevant prior art of record, Bae with Leow combination, teaches 
utilizing the memory and CPU for determining a germ in each of a plurality of images, the germ 
containing a region of interest; utilizing the memory and CPU for defining a support in each of 
the video segments, wherein the support is the video segment less the germ; utilizing the memory 
and CPU for separating the germ from the video segments; utilizing the memory and CPU for 
laying out the germs on a canvas, wherein the canvas is partitioned into disjoint areas 
corresponding to the shape of the germs (see Final Rejection on 8/24/09). 

Applicant's claimed invention distinguishes over the Bae with Leow combination by 
utilizing the memory and CPU for computing boundary curves between the germs, 
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wherein the boundary curves between the germs are defined using one or more algorithm 
selected from the group consisting of the distance between a point and the closest border of the 
germ, the distance between a point and the center of a germ and the distance between a point and 
the size of the germ; utilizing the memory and CPU for defining a space between the boundary 
curves; and utilizing the memory and CPU for filling in the space of the canvas, wherein filling 
in the space of the canvas includes laying out one or more portions of the supports by assigning a 
pixel value of a point in the space from the same value as the corresponding pixel of the germ's 
support when this point is nearest the germ, and only when the germ's support does not 
encompass the point assigning pixel values of a support of a neighboring germ based on a 
distance from the point to the neighboring germ, wherein a point between the boundary curves is 
assigned an average value of nearby point values only if no support includes the point, to 
generate a highly condensed visual summary of the plurality of video segments. 

Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany the issue 
fee. Such submissions should be clearly labeled "Comments on Statement of Reasons for 
Allowance." 

Conclusion 

5. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to EDWARD PARK whose telephone number is (571)270-1576. 
The examiner can normally be reached on M-F 10:30 - 20:00, (EST). 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Samir Ahmed can be reached on (571) 272-7413. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Edward Park 
Examiner 
Art Unit 2624 

/Edward Park/ 
Examiner, Art Unit 2624 
/Brian Q Le/ 

Primary Examiner, Art Unit 2624 



